Data Integration using Amazon AppFlow

Amazon AppFlow is a data integration service that lets you load data from SaaS applications such as Salesforce or ServiceNow into Amazon S3 data lake.

Prerequisites

To use Amazon AppFlow for data integration, you must complete the following prerequisites:

  • Configure an Amazon AppFlow instance from Cloud Platform Tools & Technologies.

  • Configure Salesforce or ServiceNow to use as a data source.

To create a data integration job using Amazon AppFlow

  1. Click the Amazon AppFlownode in the data integration stage of the pipeline, and click Create Job.

  2. Complete the following steps to create the job:

Job Name - Provide an appropriate name for the data integration job.

Configure the source node:

  • Review the configuration details of the source node. Based on the data source you select, the configuration details are fetched from the data source stage.

  • Click Next.

Configure the target node:

  • Datastore - select the S3 datastore that you want to configure.

  • Choose Target Format - select one the following target formats:

    • Source Data Format- select this option if you want to maintain the data format in the target, similar to that in the source.

    • Parquet - select this option if you want to use parquet format for target data.

    • Delta Table - select this option if you want to create a table with delta data.

  • Target Folder - select a target folder on S3.

  • Target path - provide a folder name that you want to append to the target folder. This is optional.

You can review the final path of the target file. This is based on the inputs that you provide.

  • Click Next.

Tags help you manage, identify, organize, and search for resources. Add the tags in key value pairs as required.

  • Click + New Tag.

  • Enter the key and value.

  • Click Next.

SQS and SNS

  • Configurations - Select an SQS or SNS configuration that is integrated with the Lazsa Platform

  • Events - Select the events for which you want to enable SQS or SNS queues.

    • Select All

    • Node Execution Failed

    • Node Execution Succeeded

    • Node Execution Running

    • Node Execution Rejected

  • Event Details - Select the details of the events for which notifications are enabled.

  • Additional Parameters - provide any additional parameters to be considered for SQS and SNS queues.

 

Related Topics Link IconRecommended Topics What's next? Databricks Templatized Data Integration Jobs